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Abstract 

Substantial  expectations  have  been  set  about  the  effectiveness  and  role  that  high  level  Interactive 
Electronic  Technical  Manuals  (IETMs)  for  performance  aiding  will  play  in  enabling  lesser 
skilled  U.S.  Navy  maintainers  to  perform  their  jobs.  This  empirical  study  about  the  design  and 
effectiveness  of  high  level  IETMs  provides  baseline  and  comparative  data  about  two  high  level 
IETM  interfaces  used  for  one  F/A-18  aircraft  maintenance  task.  Eight  maintainers  whose 
experience  levels  varied  from  new  (less  than  one  month)  to  very  experienced  (more  than  eight 
years)  participated  in  this  study.  Both  baseline  and  “with  IETM”  data  collection  efforts  occurred 
at  China  Lake  Naval  Weapons  Station.  Study  results  include  performance  data  and  insights  about 
needed  improvements  to  attain  correctness,  speed,  and  ease  of  use  in  information  search, 
navigation  and  magnification  activities.  Differences  in  expert  and  novice  preferences  and 
performance  were  documented  to  inform  future  adaptive  interface  design  efforts.  The 
maintainers  who  participated  were  unanimously  enthused  about  the  possibility  of  having 
improved  IETMs  on  small  mobile  computers  for  performance  aiding  in  the  near  future. 


This  research  was  partially  supported  by  the  Office  of  Naval  Research  (ONR  Final  Report 
#N00014-00- 1-0727,  November  30,  2000).  The  views  and  conclusions  contained  in  this 
document  are  those  of  the  authors  and  should  not  be  interpreted  as  representing  the  official 
policies,  either  expressed  or  implied,  of  the  Office  of  Naval  Research  or  the  U.S.  Government. 
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I.  Introduction 


As  noted  at  the  U.S.  Navy  Workshops  on  the  Generalized  Maintainer,  aircraft  carrier 
and  aircraft  maintenance  for  the  U.S.  Navy  is  changing  today  in  three  key  operational 
aspects:  (1 )  paper  documentation  is  being  replaced  with  Interactive  Electronic  Technical 
Manuals  (lETM’s),  (2)  the  number  of  personnel  performing  the  work  is  shrinking,  and  (3) 
the  amount  of  specialized  training  maintainers  receive  to  keep  aircraft  and  ships  at  full 
operational  readiness  is  being  reduced.  These  changes  are  also  occurring  in  the  other 
military  services  and  the  civilian  aircraft  sectors. 

The  Navy  and  all  the  military  services  are  challenged  to  make  these  changes  while 
meeting  an  ambitious  time  schedule.  The  study  we  report  on  in  this  document  is 
directly  applicable  to  attainment  of  the  Navy’s  objective  to  improve  training  and 
performance  aiding.  Currently,  there  are  substantial  unknowns  about  both  the  effective 
authoring  of  lETMs  and  their  use  in  maintenance  training  and  performance  aiding.  Of 
greatest  importance  is  the  need  for  knowledge  about  how  lETMs,  communications 
capabilities,  and  computer  technology  can  support  Navy  maintainers  to  effectively  use 
lETMs  while  in  combat  and  other  operational  situations.  This  basic  empirical  study  and 
others  like  it  are  essential  for  the  Navy  to  have  underway  now  so  that  objectives  for 
improved  training  and  performance  aiding  can  be  accomplished  without  adversely 
impacting  force  readiness. 

The  next  section  (II)  of  this  report  summarizes  insufficiencies  of  current  lETMs  for 
performance  aiding  identified  in  this  study  and  a  description  of  the  IETM  problems  that 
fall  within  the  scope  and  focus  of  this  study.  Section  III  presents  the  study  procedures 
and  design.  Section  IV  presents  the  results  of  this  effort.  Section  V  presents  the 
conclusions  and  recommendations  from  this  study.  Appendix  I  contains  the  initial 
demographic  data  collection  instrument  and  Appendix  II  has  the  post-trial  self-report 
questionnaire  used  in  the  study.  This  document  is  accompanied  by  a  CD-ROM 
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containing  video  clips  of  maintainers’  performance  as  referenced  in  this  report.  The 
videos  are  formatted  as  Quick-time  movies  for  viewing  on  either  PC  or  Mac  platforms. 

II.  Insufficiencies  of  Current  IETM  Interfaces 

Numerous  problems  with  currently  available  Level  IV  lETMs  were  identified  during  the 
initial  phase  of  this  project.  First,  there  was  no  attention  to  adaptive  interfaces  that 
would  take  into  account  the  varied  experience  levels  of  users.  Second,  in  spite  of  Navy 
Standards  to  promote  effective  IETM  development,  it  appears  that  these  Standards, 
e.g.,  MIL-PRF-87268A  (1 997)  do  not  result  in  development  of  lETMs  that  take  into 
account  several  crucial  aspects  of  human  usability.  Third,  the  design  of  current  lETMs 
requires  that  maintainers  use  precisely  correct  nomenclature  or  the  system  does  not 
find/linkthem  to  needed  information.  Fourth,  the  use  of  lETMs  in  Personal  Computers 
(PCs)  often  involves  problems  related  to  hardware,  display  technology,  Input/Output 
devices,  and  Operating  Systems.  It  is  almost  impossible  to  divorce  the  effects  of  the 
overall  computer  system  from  an  understanding  of  IETM  design  and  usability  issues. 

For  this  study  we  did,  however,  focus  on  five  key  aspects  of  interface  design  that  are 
described  below.  Then  we  detail  the  prioritization  of  these  aspects  that  we  addressed  in 
the  interfaces  tested  in  this  study.  The  solutions  we  prototyped  and  tested  are 
described  in  Section  III  of  this  report. 

1.  Windows-related  Features 

There  are  a  number  of  interface  elements  that  are  inherited  by  default.  For  experienced 
computer  users,  they  are  expected  features.  However,  for  IETM  users,  they  are 
unnecessary.  Given  the  low  level  of  computer  literacy  found  with  the  participants  in  this 
and  several  other  recent  user  tests  conducted  with  military  maintainers,  Windows- 
related  features  present  a  serious  potential  source  of  difficulty  and  error. 

When  designing  an  interface  for  novice  users,  it  is  very  important  to  provide  only  the 
interactive  options  necessary  for  the  task  at  hand.  The  more  options  available  to  the 
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user,  the  easier  it  will  be  for  the  user  to  become  confused  and  select  the  wrong  option 
(e.g.  minimizing  the  window),  leading  to  confusion  and  frustration. 

To  perform  the  tasks  outlined  in  the  lETMs,  the  users  do  not  need  access  to  the 
operating  system  (and  probably  would  be  confused  if  they  found  themselves  there).  The 
"Start  bar,"  therefore,  is  a  large  source  of  confusion,  even  if  it  only  pops  up  when  the 
cursor  is  at  the  bottom  of  the  screen.  It  can  and  should  be  turned  off.  The  title  bar  and 
the  controls  it  contains  are  also  unnecessary.  In  some  lETMs  it  states  the  system  and 
procedure  being  viewed.  This  is  important  information,  but  the  title  bar  is  not  the  best 
place  for  it,  as  putting  it  there  disconnects  it  from  the  IETM  content  it  is  labeling.  A  third 
concern  is  the  “minimize”  and  “maximize”  buttons  that  also  have  the  potential  to  lead  to 
confusion.  A  fourth  feature,  the  “close”  button  would  be  better  implemented  as  a  "quit" 
button  placed  with  the  other  controls  within  the  lETMs.  Finally,  interface  menus  often  do 
not  fit  within  the  lETM’s  interface.  They  are  too  small  and  crowded  in  their  current 
placement,  and  for  the  most  part,  they  contain  the  functions  accessible  through  buttons. 
Any  functions  that  are  only  found  within  the  menus,  such  as  "Help,"  can  be  integrated 
with  the  rest  of  the  interface  as  easier  to  use  buttons. 

These  are  major  problems,  but  they  may  be  difficult  to  fix,  as  they  are  due  to  the  nature 
of  the  operating  system  itself.  If  such  changes  can  be  made  in  any  newly  developed 
IETM  system,  they  should  be  given  a  high  priority. 

2.  Information  Display 

The  information  display  area  of  the  interface  is  divided  into  two  sections,  one  for  the 
textual  steps  of  the  procedure,  and  one  for  the  accompanying  illustrations.  They  are 
divided  by  a  vertical  rule,  which  changes  position  depending  upon  the  size  of  the 
illustration.  This  can  be  improved  by  either  giving  the  user  control  over  its  placement,  or 
making  it  immovable.  Whatever  the  case,  the  buttons  that  appear  on  this  display  are  not 
now  and  should  be  coupled  to  the  area  they  affect. 
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In  the  left  section,  lETMs  now  often  show  only  one  procedural  step  at  a  time,  giving  the 
user  no  context  within  the  procedure  as  a  whole.  Occasionally  a  step  will  have  substeps 
indented  beneath  it.  When  this  is  the  case,  the  current  substep  has  a  black  rectangle 
outlining  it.  The  user  is  expected  to  continue  using  a  “next”  arrow  to  progress,  or  may 
click  on  the  substep.  There  are  much  better  methods  of  indicating  which  step  the 
maintainer  is  currently  using.  There  is  plenty  of  space  on  a  screen  to  show  more  than 
one  step  at  a  time.  Also,  there  could  be  an  indication  of  which  steps  have  illustrations, 
requiring  the  user  only  to  click  on  the  ones  he  wants  to  see,  rather  than  having  to  click 
between  each  step. 

In  some  lETMs  the  illustrations  in  the  right  frame  are  labeled  in  a  poor  fashion.  For 
example  in  one  system,  a  large  gray  circle  represents  each  label  with  an  arrow 
extending  to  the  relevant  part.  When  the  cursor  moves  over  the  circle,  the  text  label 
appears.  The  reasoning  behind  this  implementation  was  to  allow  more  labels  to  fit  on 
the  screen  than  would  be  possible  otherwise.  However,  an  examination  of  this  IETM  did 
not  readily  reveal  any  instances  of  an  illustration  within  the  lETMs  with  enough  labels  to 
require  this  treatment.  Even  if  some  exist,  there  is  a  better  way  of  handling  the  situation. 
Among  the  many  problems  with  this  roll-over  method  are:  (1)  “rollovers”  don't  work  with 
pen-based  input,  or  any  other  input  that  does  not  use  a  free  roaming  cursor;  (2)  the 
arrow  often  gets  lost  in  the  diagram,  making  it  hard  to  find  the  component  or  label  in 
question  (this  is  compounded  when  the  arrowhead  is  larger  than  its  target);  (3)  when 
the  user  magnifies  the  illustration,  the  labels  are  off  of  the  screen,  often  not  able  to  be 
seen  at  the  same  time  as  the  components  they  point  to.  Fourth,  arrows  point  to  one 
specific  point  on  the  components  they  target.  Labels  should  be  able  to  refer  to  objects  in 
their  entirety.  Finally,  all  of  the  labels  link  to  something  when  clicked  on.  Sometimes 
they  link  to  another  diagram,  and  in  these  cases,  rather  than  labeling  something,  they 
cryptically  say  something  like  “diagram.”  More  often  than  not,  they  bring  up  a  new 
screen  that  is  empty  aside  from  the  words  in  the  label  itself.  This  is  not  helpful  to  users. 

In  many  lETMs,  a  three-dimensional  arrow  labeled  “FWD”  accompanies  each  of  the 
illustrations.  This  arrow  is  supposed  to  orient  the  illustration  to  the  plane,  pointing  out  in 


7 


which  direction  the  front  of  the  plane  is  located.  A  much  more  intuitive  method  would  be 
to  have  an  icon  of  a  plane  in  the  proper  orientation. 

There  are  numerous  changes  that  could  be  made  to  improve  information  display.  The 
high  priority  changes  needed  include:  (1 )  fixing  the  vertical  rule  that  separates  the  two 
areas,  (2)  displaying  multiple  steps  so  the  user  can  get  a  sense  of  where  they  are  in  the 
overall  task,  (unfortunately,  this  is  tied  into  the  step  navigation  mechanism,  which 
makes  for  a  larger  effort  to  fix),  and  (3)  changing  the  labeling  of  the  illustrations  (this 
may  require  manipulation  of  the  content  to  some  extent).  Making  these  changes  should 
enable  attainment  of  the  main  goal  of  lETMs,  to  display  content  effectively. 

3.  Buttons 

Five  critical  problems  were  identified  with  buttons  on  the  interfaces  we  were  able  to 
examine.  First,  the  button  size  required  in  Level  IV/V  lETMs  is  too  small.  Second,  their 
placement  often  is  too  close  together.  Third,  some  buttons  seem  to  have  been 
scattered  across  the  top  of  the  screen  at  random,  rather  than  having  an  organization 
related  to  users’  mental  or  task  models.  Fourth,  some  of  the  icons  are  indecipherable; 
their  symbolism  was  impossible  for  both  researchers  and  maintainers  to  determine 
initially.  Fifth,  in  some  interfaces  when  the  functions  that  buttons  access  are 
unavailable,  the  buttons  disappear  from  the  interface,  giving  no  indication  that  they 
exist. 

There  are  two  buttons  that  we  term  the  "Fast  Forward"  and  "Rewind"  buttons,  as  the 
double  arrow  icons  take  their  cue  from  those  on  a  cassette  player.  We  don't  know  for 
certain  what  these  buttons  do.  Our  best  guess  was  that  they  allow  the  user  to  jump  from 
one  procedure  to  another  in  whatever  method  of  serial  organization  the  creators  have 
decided  to  use  (most  likely  a  numerical  reference).  If  this  is  their  purpose,  we  believe 
that  they  are  not  necessary  or  useful.  If  the  user's  task  is  to  disassemble  a  component, 
there  doesn’t  appear  to  be  is  any  reason  he  would  want  to  jump  to  a  random  procedure 
from  the  one  he  is  using.  Another  possible  purpose  would  be  to  jump  directly  to  the 
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beginning  or  end  of  the  procedure.  There  are  more  useful  ways  of  implementing  such 
navigation. 

Probably  the  most  important  controls  in  the  entire  interface,  the  “previous”  and  “next” 
buttons  deserve  a  little  more  prominence  within  the  interface  than  they  are  currently 
given  in  many  lETMs.  Warnings,  cautions,  and  alerts  show  up  as  buttons  after  the  user 
has  clicked  through  them.  However,  there  is  no  visual  connection  between  them.  There 
are  better  methods  of  doing  this  as  well. 

There  are  two  types  of  magnification  in  Level  IV  lETMs.  The  user  may  click  one 
magnification  button,  which  brings  up  a  magnified  view  in  a  separate  window  controlled 
by  moving  the  cursor  over  the  original  illustration.  Clicking  the  other  magnification  button 
turns  the  cursor  into  a  magnifying  glass,  which  can  be  used  to  magnify  the  illustration  by 
clicking  on  it.  The  two  icons  look  almost  identical.  There  also  are  better  ways  of 
handling  magnification  and  one  solution  was  tried  in  this  study  and  is  described  in 
Section  III. 

There  are  four  other  buttons  in  the  example  procedures  we  reviewed  for  which  we  do 
not  know  the  functions.  One  has  a  drawing  of  a  bolt,  one  has  a  poor  drawing  of  a 
wrench,  and  the  other  two,  we  cannot  discern.  The  use  of  ambiguous  symbols  on 
buttons  is  a  serious  problem  for  users. 

Changes  to  buttons,  such  as  size,  placement,  and  replacement  of  icons  should  be  trivial 
to  accomplish  and  will  greatly  improve  usability.  Changes  requiring  greater  effort  would 
be  an  overhaul  of  the  step  navigation  and  a  new  method  for  displaying  and  retrieving 
warnings,  cautions,  and  notes.  Navigation  is  arguably  the  most  important  interface  role 
in  lETMs,  so  this  feature  should  be  given  highest  priority  in  improving  lETMs.  Warnings 
also  are  a  very  important  part  of  the  content,  and  should  therefore  be  given  a  relatively 
high  priority  as  well. 
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4.  Warnings,  Cautions,  &  Notes 


In  most  current  Level  IV  lETMs  warnings,  cautions,  and  notes  appear  in  a  window 
overlaying  everything  else  on  the  screen.  This  requires  the  user  to  close  the  window 
before  proceeding.  While  this  practice  is  in  compliance  with  current  military  standards, 
we  believe  there  are  better  methods,  which  we  addressed  in  one  of  the  prototypes 
tested  in  this  study.  Current  visual  representations  also  contain  some  Windows 
“baggage”.  For  example,  the  title  bar  says,  "ALERT  WINDOW."  This  is  redundant, 
since  directly  below  that  it  labels  the  window  as  a  warning,  caution,  or  note.  Also  in  the 
title  bar  is  the  close  box.  This  is  also  redundant,  since  there  is  an  acknowledgement 
button  provided  at  the  bottom  of  the  window  (placed  there  with  the  expectation  that  the 
user  will  read  the  contents  of  the  window  before  closing  it).  We  suggest  that  the  entire 
title  bar  be  removed  from  such  lETMs.  Though  it  does  currently  serve  one  positive 
purpose,  and  that  is  to  let  the  user  move  the  window. 

While  the  use  of  color  is  a  good  idea,  its  current  implementation  is  lacking.  The  fully 
saturated,  high  contrast,  striped  borders  of  caution  and  warning  windows  not  only  do 
their  job  of  attracting  the  user's  attention,  they  actually  distract  users  from  reading  the 
contents  of  the  window. 

From  what  we  see,  there  appears  to  be  a  disconnect  between  what  the  client  expects 
(what  the  military  standards  setters  specified)  and  what  the  user  needs.  It  would  be 
desirable  to  develop  a  compromise  here  that  works  for  everyone. 

5.  Typography 

The  text  within  high  level  lETMs  is  not  benefiting  from  the  typographical  evolution  of  the 
past  hundred  years.  There  are  many  variables  that  can  make  lETMs  much  easier  to 
read  and  use.  Type  size,  color,  and  weight,  when  used  effectively,  result  in  a 
typographic  hierarchy  that  allows  the  reader  to  quickly  scan  the  screen  and  easily  find 
the  information  he  is  looking  for.  Type  size,  leading  (line  spacing),  typeface  choice,  line 
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length,  and  other  formatting  considerations  can  allow  the  user  to  read  text  more  easily 
and  quickly.  For  example,  ALL  CAPS  has  been  used  extensively  in  lETMs.  Text  set  in 
all  caps  is  difficult  to  read  due  to  the  lack  in  variation  of  letter  height  that  allows  readers 
to  subconsciously  recognize  words.  Unfortunately,  the  developers  of  the  military 
standards  appear  to  be  unaware  of  a  substantial  literature  (see  the  annotated 
bibliography  produced  by  the  study  team)  and  set  of  practices  that  could  greatly 
improve  readability  of  lETMs.  Several  of  these  typographical  advances  are  included  in 
one  of  the  interfaces  we  tested. 

Prioritizing  IETM  Improvements 

Based  on  the  literature  review  and  expertise  of  the  interdisciplinary  team  working  on  this 
research  effort,  we  determined  that  several  improvements  could  be  made  and  tested. 
Because  of  the  lengthy  list  of  candidate  changes  and  the  limited  resources  of  this  ONR 
study,  the  team  prioritized  changes  using  the  following  considerations:  (1)  the  expected 
degree  of  usability  improvement,  (2)  time  and  effort  required  to  implement,  and  (3)  the 
difficulty  of  identifying  and  implementing  a  sound  solution.  The  priorities  determined 
were: 

a.  Procedure  display  and  navigation  are  the  primary  functions  of  the  interface,  and 
would  most  likely  provide  the  largest  degree  of  usability  improvement. 

b.  The  typography,  general  layout,  and  button  improvement  are  technically  trivial 
changes  with  a  large  return  on  investment  for  usability  improvement. 

c.  Windows  “baggage”  provides  a  major  opportunity  for  user  confusion,  but  may  be  the 
hardest  features  to  change.  Thus,  these  issues  were  minimally  addressed  in  this  effort. 

d.  Illustration  labeling  may  be  easy  to  design,  but  time  consuming  to  implement.  The 
improvement  in  usability  would  be  important.  Thus,  a  limited  alternative  to  the  currently 
available  labeling  was  tested  in  this  effort. 
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e.  Warnings,  cautions,  and  notes  may  be  the  most  difficult  part  to  design,  as  the  users 
are  at  odds  with  the  clients  and  standard  setters  on  this  matter.  It  would  also  be  time 
consuming  and  technically  difficult.  However,  given  the  potential  for  improving  the  user's 
experience  a  limited  test  of  an  alternative  design  was  done  for  the  mocked-up  interface 
in  this  study. 

III.  Study  Description 

This  empirical  study  began  in  May  2000  with  an  extensive  literature  review  (Siegel,  J., 
Hyder,  E.  and  Nawrocki,  E.,  2000)  to  determine  the  state  of  advanced  lETMs  for 
performance  aiding  and  to  inform  the  prototype  development  and  testing  of  lETMs  for 
F/A-1 8  maintainers.  Major  study  tasks  included: 

1 .  Selection  of  a  candidate  F-1 8  maintenance  task  for  the  study. 

2.  Preparation  of  field  data  collection  instruments  (demographic,  self-report  and 
observational  questionnaires  and  checklists). 

3.  Selection  and  specification  of  a  set  of  measures  to  be  used  in  study. 

4.  Documentation  of  field  research  design. 

5.  Preparation  of  navigational  and  procedural  mock-up  alternatives. 

6.  Selection  and  preparation  of  hardware. 

7.  Collection  of  baseline  performance  data  for  the  task  (without  lETMs). 

8.  Collection  of  the  same  F-1 8  maintainers’  task  performance  using  Level  4/5  lETMs 
including  documentation  of  their  navigational  and  interaction  preferences. 
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For  tasks  7  and  8  above,  the  study  team  made  use  of  self-report,  observational,  and 
think  aloud  protocol  data  that  was  captured  along  with  each  participant’s  actual  task 
performance  via  video  and  audio  taping.  As  noted  earlier  in  this  report,  examples  of 
video  clips  accompany  this  report  in  a  CD. 

Study  procedures  and  design 

The  site  selected,  China  Lake  Naval  Weapons  Station  afforded  us  the  opportunity  to 
have  actual  F/A-18  maintainers  as  study  participants.  We  conducted  most  of  the 
research  during  their  training  days  so  as  to  minimize  any  impact  to  normal  operations. 
The  cooperating  officials  assured  the  study  team  that  we  would  have  access  to  at  least 
six  to  eight  maintainers  who  varied  in  their  experience  levels  with  aircraft  maintenance 
and  experience  with  F/A-18  aircraft  specifically.  Also  we  were  afforded  opportunities  to 
try  out  the  lETMs  both  on  the  flight  line  (for  all  baseline  instances)  and  in  the  hangar  (for 
all  “with  IETM”  trials). 

Task  selection  was  done  to  provide  both  a  practice  and  actual  task  that  had  sufficient 
complexity  to  test  the  effects  of  providing  online  documentation  that  maintainers  would 
truly  need  to  use  to  complete  their  activities.  Also  we  selected  tasks  that  would  require 
some  mobility  so  we  could  compare  the  impact  of  having  the  IETM  be  very  portable  in 
contrast  to  current  carrying  of  large  binders  of  paper  documentation.  This  study  used 
the  paper  documentation  during  the  baseline  study  and  then  one  month  later  used  an 
actual  level  IV  IETM  and  a  realistic  mock-up  of  alternative  navigation  and  search 
approaches.  The  tasks  used  in  both  the  baseline  and  “with  IETM”  parts  of  the  study 
were:  (1)  a  practice  task  of  several  steps  in  the  Inertial  Navigation  Unit  removal  task 
(using  paper  and  online  versions  of  A1  -FI  8AC-730-300,  Change  7  - 1  April  1 999  )  and 
(2)  for  the  actual  trial,  the  Inertial  Navigation  System  (INS)  Bit  Test  (using  A1-F18AC- 
730-200,  Change  1 1 ).  Five  participants  carried  out  these  tasks  on  the  same  F/A-1 8 
model  C  aircraft  for  the  baseline  study  and  seven  participants  (including  four  of  the 
original  participants)  did  so  on  the  same  model  D  aircraft  for  the  “with  IETM”  condition. 
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ONR  had  a  unique  opportunity  to  link  this  study  to  more  applied  work  since  we  had 
access  to  and  selected  cutting  edge  mobile  computers  developed  by  IBM  (Ditlea,  2000) 
with  VGA  quality  head-worn  displays  and  Level  4/5  lETMs  for  F/A-18  aircraft. 

Originally,  we  intended  to  use  ViA  Inc.’s  wearable  computers  with  a  visual  display 
produced  by  MicroOptical  Corporation.  However,  due  to  difficulties  with  MicroOptical  in 
shipping  working  systems  within  the  timeframe  of  this  study,  use  of  the  IBM  devices 
became  the  best  option.  The  technical  specification  for  this  wearable  computer  is: 
Pentium  233  MMX  processor,  64  MB  RAM  and  680  MB  hard  drive  equivalent  to  an  IBM 
ThinkPad  560X,  but  weighing  just  400  grams  and  measuring  just  26  x  80  xl  20  mm.  The 
IBM  Watson  Center  designed  head-worn  computer  display  used  is  a  small  liquid  crystal 
chip  with  an  image  transmitted  by  a  prism  and  redirected  to  the  user’s  eye.  Due  to 
magnifying  optics  in  the  device,  it  appears  to  the  user  that  they  are  reading  a  full-sized 
desktop  computer  screen  with  VGA  resolution.  The  I/O  device  used  in  this  study  was  a 
dial  developed  at  CMU  to  enable  easy  almost  hands-free  navigation  as  shown  in  Figure 
1 .  The  users  rotated  the  dial  through  options  for  each  action  to  be  taken  within  in 
interface  and  then  depressed  a  button  on  the  edge  of  the  dial  to  select  an  option.  Note 
that  the  head-worn  display  was  attached  to  the  maintained’  cranial  cap  for  this  study. 

The  intended  design  for  this  evaluation  was  a  repeated  measures,  within  subjects, 
comparison  of  performance  and  preferences.  Every  effort  was  made  to  have  as 
controlled  an  evaluation  effort  as  possible.  Due  to  operational  priorities,  just  five 
participants  were  available  for  the  study  in  the  baseline  case.  Three  additional 
maintainers  were  available  for  the  “with  IETM”  activities  and  one  of  the  original  five 
participants  was  unable  to  take  time  from  his  operational  duties.  A  one-month  long 
interval  was  used  between  the  baseline  and  “with  IETM”  treatments  to  mitigate  against 
any  learning  effects  of  participants.  Each  participant  performed  the  baseline  task  in  late 
August,  2000  and  the  “with  IETM”  task  in  early  October,  2000.  The  two  IETM  interfaces 
tested  were  varied  randomly  in  the  order  of  use  and  one-half  of  each  task  (practice  and 
INS  Bit  test)  was  performed  using  each  interface. 
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Figure  1 :  Maintainer  wearing  vest  containing  IBM  wearable  computer  with  dial 
for  I/O  and  head-worn  display  on  cranial  cap  in  front  of  right  eye 

IETM  Interfaces  Tested 

As  noted  previously,  two  different  IETM  interfaces  were  tested  in  this  study.  Interface  A 
is  a  Windows/Web  browser  design  that  is  typical  of  interfaces  in  use  today,  as  shown  in 
Figure  2  below.  We  also  chose  this  interface  because  it  has  an  appearance  similar  to 
the  paper  documentation.  The  procedure  is  listed  on  the  left-hand  side  and  the 
associated  figure  is  displayed  on  the  right-hand  side.  Users  could  change  the  screen 
layout  by  clicking  on  the  appropriate  icons  in  the  upper  right-hand  corner  of  the  display. 
The  viewing  options  participants  had  were:  (1)  one  screen  with  no  division,  (2)  the 
current  two-part  display  with  a  vertical  division  or  (3)  a  two-part  display  with  a  horizontal 
division.  Note  that  this  interface  uses  a  smaller  font  than  Interface  B  and  contains  some 
icons  and  options  (e.g.,  tabs  at  the  top)  whose  meaning  and  functions  may  not  be  clear 
to  users.  For  instance,  the  “Program”  tab  may  not  be  a  topic  that  aircraft  maintainers 
would  expect  to  encounter  and  they  might  not  realize  that  this  part  of  the  prototype 
contains  a  function  to  save  a  session  (in  case  they  need  to  stop  before  completing  a 
procedure). 
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Figure  2:  Screen  shot  of  Interface  A,  Web-browser  IETM  interface  for  the  Inertial 
Navigation  Bit  Test 

Interface  B  was  mocked-up  for  this  study  and  is  highly  graphical.  An  example  screen  is 
shown  in  Figure  3.  Following  is  a  description  of  the  design  solutions  incorporated  in 
Interface  B  to  address  the  insufficiencies  of  the  current  lETMs.  We  note  at  the  outset 
that  some  of  the  decisions  for  this  interface  were  associated  with  the  fact  that  it  was 
specifically  designed  for  the  CMU  dial. 
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Procedure  Display  and  Navigation 

We  gave  the  user  context  within  the  procedure  by  displaying  more  than  one  step  at  a 
time.  First  we  created  three  sections  representing  each  section  of  the  procedure.  Then 
we  divided  each  section  into  "pages."  Each  page  contained  the  number  of  steps  that 
could  fit  on  screen  at  one  time. 

The  navigation  frame  at  the  top  of  the  screen  displays  an  icon  for  each  page  in  the 
procedure.  A  user  can  dial,  using  the  top  navigation  frame,  from  one  page  to  the  next, 
changing  the  set  of  steps  seen  in  the  lower  frames.  An  expert  user  could  perform  the 
entire  procedure  in  this  fashion  if  he  did  not  need  the  illustrations  and  other  information 
provided  on  a  step-by-step  basis.  A  novice,  or  less  experienced  user,  could  dial 
through  the  lower  frames,  selecting  each  step,  to  get  more  detailed  information  about 
that  step  (including  illustrations  and  links  to  other  information).  The  user  can 
immediately  see  where  he  is  in  the  procedure  by  glancing  at  the  navigation  screen  at 
the  top  of  the  page. 

Typography 

We  used  a  1 3  point  sans  serif  font,  Arial ,  with  20  point  leading  for  several  reasons.  1 )  It 
is  easily  readable  in  the  headworn  display  selected  for  the  testing.  2)  This  font 
selection  allows  a  reasonable  number  of  lines  of  text  to  fit  in  the  frame.  3)  The  font 
accommodates  a  comfortable  line  length  even  with  the  screen  split  vertically  to  provide 
room  to  display  illustrations. 

Background  shading  is  used  on  alternate  steps  to  visually  separate  the  steps.  This  is 
also  useful  to  indicate  which  icons  are  associated  with  which  steps,  as  there  are  often 
multiple  icons  associated  with  a  single  step. 

Buttons/Icons 

Icons  were  designed  to  be  easily  recognizable  and  meaningful  to  the  intended  users. 
Color  is  used  to  indicate  which  items  on  the  screen  can  be  selected,  as  well  as  to 
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separate  the  icons  from  the  illustrations,  which  often  contain  buttons  and  other  controls 
in  the  cockpit. 

There  were  two  types  of  icons  in  this  interface.  One  type  of  icon  was  related  to  the 
specific  procedure  being  performed  (e.g.,  cautions,  illustrations).  These  icons  were  a 
graphical  representation  of  the  type  of  action/information  displayed  within  a  circle.  The 
second  type  of  icon  indicated  links  to  other  procedures  and  is  given  a  different  visual 
treatment  to  differentiate  this  type  of  icon  from  the  other  icons.  Icons  that  indicate  links 
are  graphically  represented  as  a  notebook,  without  the  surrounding  circle. 

Because  the  screen  real  estate  is  so  small,  and  the  images  can  be  extremely  detailed, 
we  provided  a  method  for  viewing  the  image  utilizing  most  of  the  screen.  Selecting  and 
pressing  the  appropriate  icon  causes  the  image  frame  to  expand  to  the  left,  covering  the 
text  frame,  and  allowing  the  user  to  see  more  of  the  image  at  one  time.  Magnification 
and  scrolling  can  also  be  performed  in  this  mode.  Another  icon  allows  the  user  to  go 
back  to  the  normal  view. 

Windows-related  Features 

This  issue  was  easily  addressed  in  our  prototype,  as  our  prototyping  tool,  Macromedia 
Director,  allows  the  presentation  to  take  over  the  screen,  hiding  all  Operating  system 
level  widgets.  There  was,  therefore,  no  interface  components  displayed  onscreen  other 
than  those  that  were  a  part  of  the  IETM  prototype. 

Alerts  -  Warnings,  cautions,  notes 

Alerts  are  attached  to  individual  steps  as  icons.  As  the  user  goes  through  the  procedure 
step-by-step,  when  the  icon  is  selected  the  alert  displays  in  the  right  hand  frame  and 
cannot  be  skipped.  However,  the  alerts  do  not  obscure  the  procedure  when  they  are 
displayed,  as  they  do  in  other  lETMs. 
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INS  Ground  Initiated  BIT  Test 
Preliminary  Instructions 


®Do  nose  wheelwell  digital  display 

indicator  built-in  test/reset  procedure. 

lO  Apply  electrical  power. 


On  GND  PWR  control  panel  assembly, 
set  and  hold  1  and  2  switches  to  B  ON 
for  3  seconds.  Switches  latch  and 
remain  on 

On  left  digital  display  indicator  (LDDI), 
set  power  switch  to  DAY  or  NIGHT  as 
applicable.  Allow  2  minute  warmup  and 
adjust  BRT  and  CONT  for  best  display. 
LDDI  displays  MENU  pushbutton  switch 
option. 


° 


E 


Note 


Component  locations  are  shown  in 
WP004  00. 

If  a  malfunction  occurs  during  this 
test,  make  sure  circuit  breakers 
shown  in  WP004  00  are  closed. 


Figure  3:  A  screen  shot  of  the  mocked-up  Interface  B  for  the  INS  Bit  Test  task. 


IV.  Results 


In  this  section,  we  describe  the  participants.  Then  we  provide  specific  instances  where 
the  maintainers  either  succeeded  or  had  major  difficulty  with  particular  aspects  of  the 
interfaces  followed  by  a  summary  table  of  all  problems  encountered  during  the  IETM 
trial.  Next  we  summarize  their  self-report  feedback  about  use  of  the  two  IETM 
interfaces  tested  in  the  study.  Finally,  we  summarize  the  key  aspects  of  the  interfaces 
they  preferred  associated  with  their  experience  levels. 

Demographic  data 

The  eight  study  participants  at  China  Lake,  California  had  characteristics  shown  in 
Table  1. 
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Characteristic 

Mean 

Minimum 

Maximum 

Age  (years) 

27 

20 

37 

Years  as  F/A-18  maintainer 

2  yr.  1 1  mo. 

2  mo. 

9  yr.  3  mo. 

Years  in  job  at  China  Lake 

1  yr.  6  mo. 

2  mo. 

5  yr.  2  mo. 

Years  doing  aircraft  maintenance 

6  yr. 

Iyr.1  mo. 

10  yr. 

Table  1 :  Demographic  characteristics  of  study  participants  (n  =  8) 


Six  of  the  eight  participants  received  School  House  and  Intermediate  or  Advanced 
training  for  F/A-18  maintenance.  One  participant  was  trained  to  work  on  F-1 4  Tomcats 
and  was  recently  transferred  to  China  Lake  to  work  on  F/A-18’s  but  received  no  formal 
training  for  this  model  aircraft.  The  remaining  participant  was  trained  to  work  on  SH-60 
Helicopters  and  also  received  no  F/A-1 8  training.  Participants  were  asked  to  rank  their 
aircraft  maintenance  skill  level  (using  a  scale  where  1  =  none  and  7  =  extensive).  The 
average  rating  from  the  eight  participants  was  5.5. 

Participants  were  queried  about  their  previous  computer  training  and  none  had  received 
any  formal  training  prior  to  military  service.  When  asked  about  their  “hands-on” 
computer  experience,  four  indicated  they  had  moderate  prior  experience  and  four 
indicated  they  had  none.  Seven  of  the  eight  men  had  some  prior  experience  with 
lETMs. 

Questions  regarding  vision  status  of  participants  revealed  the  fact  that  two  of  the  eight 
had  no  vision  correction,  three  men  had  corrective  lens  for  myopia  and  one  was  far¬ 
sighted.  Two  men  reported  other  vision  problems,  e.g.,  light  sensitivity. 

Specific  interface  experiences  and  problem  areas 

Three  kinds  of  maintenance  experiences  with  the  interfaces  are  characterized  here:  (1) 
navigational  challenges,  (2)  use  of  magnification,  and  (3)  experience  with  cautions. 
There  were  also  several  instances  of  hardware  problems.  So  that  readers  gain  a  more 
concrete  sense  of  the  maintainer’s  view  for  each  of  the  aspects  of  the  interfaces  we 
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tested,  we  present  a  brief  description  of  the  activity  the  user  was  doing  and  a  screen 
shot  of  what  they  saw  on  the  IETM.  The  accompanying  CD  contains  related  video  clips 
with  the  associated  “think  aloud”  monologue  for  your  information.  Then  we  provide  a 
summary  table  of  all  the  instances  of  problems  we  coded  from  the  video  and  think  aloud 
data. 

Figure  4  illustrates  the  view  each  participant  began  with  for  the  INS  Bit  Test  task  while 
using  Interface  B.  As  the  participant  turned  the  dial,  different  steps  in  the  procedure 
would  be  highlighted.  Navigational  problems  often  were  associated  with  the  I/O  device 
For  example,  an  expert  maintainer  experienced  confusion  when  he  turned  the  dial  too 
quickly  “Think  I  went  too  fast  for  the  dial.  I’m  going  through  a  bunch  of  steps  trying  to 
catch  up.  I  think  I  went  too  far.  All  right.  I’ve  got  it  now.  Next  page...”  (P001)  Other 
problems  arose  from  maintainers  being  unsure  about  how  to  go  backwards  and 
forwards.  To  get  a  clearer  sense  of  the  navigational  experience  of  participants  using 
Interface  A,  please  use  the  accompanying  CD  and  see  ONR_7_UseofLink_WUI.mov. 

In  this  instance,  the  participant  cannot  determine  how  to  get  to  the  term  “here”  to 
accomplish  a  link  and  asks  the  experimenter  for  advice  about  where  to  click. 


Figure  4:  Participant  turning  dial  to  navigate  through  INS  Bit  Test  Task  in  F/A-1 8 
cockpit  (what  he  sees  is  Interface  B  on  the  right  side  of  this  figure). 
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Magnification  was  used  with  each  of  the  interfaces  and  the  users’  views  of  magnified 
lETMs  are  shown  next  in  Figures  5  and  6.  In  general  participants  did  not  have  difficulty 
using  the  magnification  features.  As  noted  later  in  this  report,  the  participants  each 
expressed  a  desire  to  have  such  magnification  capabilities  in  lETMs. 
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Figure  5:  Participant  viewing  magnified  illustration  using  Interface  A. 

Using  Interface  A  the  participant  magnified  the  illustration  and  was  able  to  continue  the 
task  effectively.  We  also  provide  an  accompanying  video  clip  of  a  maintainer  using  this 
interface  while  being  mobile  (see  ONR_3_  Magnification  _WUI.mov).  He  used  it  so  he 
could  locate  the  correct  door  to  the  aircraft  to  perform  the  practice  task  for  this  study. 
That  participant  seemed  to  have  no  difficulty  with  simultaneously  walking,  talking  and 
seeing  the  IETM.  He  did,  however,  have  some  difficulty  seeing  the  display  without 
squinting. 
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Figure  6:  Participant  viewing  magnified  figure  using  Interface  B. 

Using  Interface  B,  the  participant  initially  could  not  see  lettering  on  the  IETM  illustration. 
He  magnified  it  to  read  the  accompanying  text  better  and  scrolled  around  to  be  sure  he 
could  see  the  entire  illustration.  This  enabled  him  to  have  confidence  that  he  knew 
everything  he  needed  to  proceed  with  the  task. 

Figure  7  shows  a  participant  and  the  screen  he  was  seeing  when  reading  a  caution 
using  Interface  A.  This  caution  is  in  accordance  with  current  IETM  standards  and 
effectively  blocks  the  maintainer’s  view  of  anything  except  the  cautionary  information. 

In  contrast,  in  Figure  8  the  maintainer  is  using  Interface  B  where  the  caution  is  still  very 
salient,  but  the  associated  procedural  steps  also  were  visible  to  the  participant.  Viewing 
the  associated  video  clip  (see  ONR_8_caution_with_note_JUI_mov.)  shows  that  as 
soon  as  he  takes  the  required  cautionary  action,  the  caution  is  cleared  from  the  screen 
enabling  him  to  continue  the  task.  In  this  instance  the  IETM  design  helped  him  to  keep 
a  connection  to  the  overall  task  structure,  but  still  forced  attention  to  the  “Caution”. 
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Figure  7:  Participant  using  Interface  A  to  view  a  caution  in  the  INS  Bit  Test  task 


Figure  8:  Participant  using  Interface  B  to  view  a  caution  in  the  INS  Bit  Test  task 


Table  2  shows  that  participants  004  and  006  had  totally  different  experiences  with 
Interface  A  and  B  (no  problems  with  one  and  many  problems  with  the  other).  All  other 
participants  had  a  slight  difference  in  performance.  However,  comparing  performance 
between  Interface  A  and  B,  there  is  no  significant  difference  in  the  total  number  of 
problems  other  participants  experienced.  There  did  appear  to  be  more  problems  with 
navigation  backward  and  forward  with  Interface  B.  Participants’  performance  was 
consistent  with  their  interface  preferences  that  they  expressed.  These  preferences  are 
reported  in  the  next  section  and  discussed  later  in  this  report. 
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Table  2:  Summary  of  participants’  problems  while  using  Interfaces  A  and  B  (n=7) 

Participants’  self-reported  IETM  experiences 

Each  of  seven  participants  completed  a  questionnaire  immediately  after  trying  out  the 
lETMs  and  also  participated  in  a  videotaped  debriefing.  All  participants  were  able  to 
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complete  their  work  and  made  mostly  positive  comments  about  the  lETMs.  They  also 
mentioned  issues  related  to  use  of  wearable  computers,  reinforcing  the  need  for  future 
studies  to  consider  the  technology  used  to  deliver  lETMs  as  well  as  the  lETMs 
themselves.  Their  responses  to  the  questions  are  summarized  in  this  section  of  the 
report. 

Participants’  were  asked  first  about  correctness  of  their  task  performance  and  then 
about  ease  of  performing  work.  They  were  fairly  confident  about  correctness  and 
indicated  that  task  completion  was  easy.  Their  specific  responses  on  these  items  were: 

1 .  How  confident  were  you  that  you  did  the  job  correctly  -  average  =  4.1 
(Scale  used:  1  =  not  sure,  5  =  very  sure) 

2.  How  easy  was  it  to  complete  the  task  -  average  =  4.4 

(Scale  used:  1  =  very  difficult,  5  =  very  easy) 

Participants  also  were  asked  whether  they  found  the  lETMs  to  be  useful  and  the 
average  response  was  equal  to  4.7  where  the  scale  used  was:  1  =  not  useful  at  all  and 
5  =  very  useful. 

We  queried  participants  about  any  problems  or  difficulties  they  experienced  and  found 
that  most  of  their  reported  problems  were  associated  with  the  wearable  system  rather 
than  the  lETMs.  None  of  the  participants  indicated  they  had  any  difficulties  reading  the 
lETMs  or  talking  with  others  while  using  the  lETMs.  Also,  none  of  them  reported  any 
problems  with  finding  the  information  they  needed.  One  participant  reported  getting 
confused  when  he  skipped  steps  he  already  knew  and  went  just  beyond  the  step  he 
expected  to  stop  at. 

Participants  were  asked  to  indicate  their  perceived  ease  of  use  or  difficulty  for  each  of 
the  two  interfaces  and  their  responses  are  in  Table  3  below. 
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Usinq  IETM  Interface  A 

Averaqes 

(1=Very  Easy;  5=Very  Difficult) 

1 .  Seeing  the  text  in  the  online  application  was 

1.14 

2.  Seeing  the  graphic  representations  was 

1.14 

3.  Using  the  index  to  find  information  was 

1.29 

4.  Finding  the  information  1  needed  to  link  to  was 

1.71 

Usinq  IETM  Interface  B 

(1-Very  Easy;  5=Very  Difficult) 

1 .  Seeing  the  text  in  the  online  application  was 

1.43 

2.  Seeing  the  graphic  representations  was 

1.14 

3.  Using  the  index  to  find  information  was 

1.71 

4.  Finding  the  information  1  needed  to  link  to  was 

2.14 

Table  3:  Ease  of  use  ratings  of  IETM  Interfaces  A  and  B 


Participants  also  had  an  opportunity  to  indicate  what  they  liked  best  and  least  about  the 
IETM  prototypes  and  their  answers  are  summarized  below  in  Table  4.  It  is  noteworthy 
that  all  of  the  comments  about  what  they  liked  least  were  related  to  the  I/O  device  or  the 
display  and  not  about  the  lETMs  themselves.  Also,  when  asked  what  they  would 
change  they  indicated  hardware,  I/O  and  display  issues. 
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Participant  # 

Comment 

001 

You  can  use  it  wherever  you  want.  No  need  to  take  out  troubleshooting 

procedures. 

002 

It  is  right  in  front  of  you  and  there  are  no  books  to  worry  about. 

003 

Easy  to  use. 

004 

The  easy  mobility  and  light  weight.  Easy  to  use  and  carry.  Able  to  move 

eyepiece  away  if  needed. 

006 

Don’t  have  to  carry  large  heavy  pubs,  and  don’t  have  to  worry  about 

whether  pubs  have  been  updated  or  not. 

007 

Being  able  to  look  at  text  while  on  the  job  or  in  the  cockpit. 

008 

Compactness,  immediate  availability  of  information. 

Table  4:  Likes  and  dislikes  about  the  IETM  prototypes. 


The  last  item  from  the  self-report  questionnaire  asked  whether  there  was  anything  else 
the  participants  wanted  to  report  and  their  answers  are  provided  here  in  Table  5. 


Participant  # 

Comment 

004 

Graphics  very  helpful,  easy  to  read  and  use.  Need  adapter  for  when  not 

wearing  cranial. 

006 

Yeah,  how  soon  it’s  coming  to  the  fleet. 

007 

Not  all  jobs  can  be  explained  on  screen  even  with  step-by-step 

procedures.  May  consider  small  animated  steps  when 

installing/removing  certain  parts  that  require  “tricks  of  the  trade” 

Table  5:  Addi 

tional  comments  about  IETM  prototypes. 

All  participants  indicated  they:  (1 )  have  a  real  dislike  for  the  way  that  cautions  and 
warnings  are  handled  in  the  lETMs  they  currently  use  and  preferred  both  Interface  A 
and  B  handling  compared  to  their  current  option;  (2)  had  a  strong  need  for  easy  use  of 
magnification,  and  (3)  prefer  having  both  the  procedural  text  and  graphic  information 
available  simultaneously.  They  noted  for  example  that  with  current  lETMs  “ . .  .the  only 
thing  that  everybody  hates  is  they’ll  bring  up  the  note  and  caution  and  you  have  to 
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acknowledge  it  to  go  to  the  next  step  and  everybody  is  really  getting  tired  of  that.” 

(P001) 

Comments  related  to  the  value  of  magnification  include:  “...there’s  certain  times  like  you 
know  if  you  need  to  see  something,  a  clamp,  a  bolt,  a  mounting  bracket,  a  pin, 
whatever,  you  can  look  at  it  to  make  sure  you  know  and  you  can  move  it  around  to  see 
how  it  should  be  fit  in  there..”  (P007)  Another  maintainer  observed  that  “  You  look  at  the 
graphics  and  it  will  show  you  what  it  looks  like,  you  can  bring  it  up  for  more  details.  It’s 
extremely  useful  being  able  to  magnify  one  picture  to  get  more  detail  out  of  it”.  (P004) 

Comments  about  seeing  both  text  and  graphics  simultaneously  on  the  screen  indicated 
that  this  was  important  in  getting  the  task  right.  For  example,  in  response  to  the 
experimenter’s  question  of  “Did  you  find  having  the  images  there  so  you  could  see  the 
picture  with  the  procedure  useful?”  Participant  002  stated  that  “...Yeh,  especially  when 
it  comes  to  looking  at  the  DDI  because  -  cause  I’m  not  really  the  greatest  guy  but  seeing 
the  DDI  display  -  so  what  I  should  be  on  is  a  good  thing.  Cause  I  was  actually  on  the 
wrong  page  one  time  and  I  had  to  correct  it.” 

Apart  from  the  features  of  lETM’s  that  all  participants  agreed  were  useful,  there  were 
specific  aspects  of  Interfaces  A  and  B  that  we  expected  would  be  preferred  and  more 
effective  for  different  users.  Analyses  of  the  data  that  give  insights  about  these  issues 
are  described  next. 

Interface  preferences  and  issues  for  novice  and  expert  maintainers 

We  analyzed  the  self-report  comments  and  debriefing  transcripts  as  well  as  the  video 
data  to  determine  whether  there  was  any  obvious  pattern  of  preference  for  the  two 
interface  options  between  novice  and  expert  maintainers.  Given  the  small  size  of  the 
study  we  would  hesitate  to  generalize.  We  do  provide  data  about  two  results  here: 
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(1)  the  difference  in  interface  preference  based  on  knowledge  level  and  mental  models 
of  technical  data  and  (2)  the  belief  that  animation  or  video  material  demonstrating 
assembly  or  disassembly  procedures  would  be  critical  to  lETMs  for  performance  aiding. 

Three  participants  (002,  004,  and  008)  preferred  the  Web-manual  design  -  two  were 
novices  and  the  third  was  an  expert,  responsible  for  Quality  Assurance  for  the 
squadron.  The  four  maintainers  who  preferred  Interface  B  (P001 ,003,  006,  and  007) 
were  fairly  experienced  (three  had  at  least  five  years  of  experience  and  one  had  eight 
years  of  experience  as  aircraft  maintainers).  The  comments  of  the  participants  suggest 
that  the  novices  mental  model  of  the  technical  information  was  the  paper  documentation 
-  so  having  an  online  representation  that  was  closest  to  that  familiar  information  format 
was  easier  for  them  to  use.  In  fact,  the  one  experienced  maintainer  who  preferred  the 
Web-manual  commented  about  his  preference  for  the  paper-like  view: 

“You  get  a  mind  set  with  the  publications  where  you  start  memorizing 
publications  and  you’re  just  able  to  pop  it  off  right  there.  While  the  other 
way  you  don’t  really  have  an  indication  of  what  pub  you’re  in... .So  if  you’re  an 
inexperienced  troubleshooter  and  you  put  this  thing  on,  you  might  not  be  able  to 
find  your  way  around  that  very  well.  The  other  way,  you  could  take  somebody 
right  out  of  A  school  that’s  got  the  pub  up  there.”  [he  pointed  to  his  head  while 
making  this  comment](P008) 

The  participants  who  preferred  Interface  B  noted  the  ability  to  have  a  sense  of  the  whole 
task  and  seemed  to  value  this  design  feature  over  the  issue  of  having  an  online 
application  that  was  similar  to  paper  documentation.  They  stated  for  example  “You 
know  the  one  with  the  icons... with  the  yellow  background.  That  one  I  liked  because  you 
had  more  information  on  one  page.  The  other  way  is  where  you  got  a  small  portion.” 
(P003)  Another  participant  stated 

“I  kind  of  like  the  second  style  -  the  icons.  Cause  it  lets  you  see  all  the  steps  at 
once  and  for  certain  individuals,  you  know,  that  are  more  experienced,  you  don’t 
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need  to  go  step  by  step  by  step  you  can  just  you  know  scroll  down  the  screen, 
instead  of  having  to  click,  click,  click.  You  can  look  at  and  read  several  steps  that 
are  you  know  inclusive  with  each  other  all  at  once.  And  then  dial  to  the  next 
step.  It  helps,  it’s  a  little  faster.  It’s  almost  as  fast  as  flipping  pages  in  a  book. 
That’s  why  a  lot  of  sailors  like  a  book  cause  you  can  flip  at  your  own  speed.  If 
you  need  to  go  slow  you  can  go  slow,  but  if  you  need  to  go  fast,  you  can  go 
faster.”  (P007) 

When  asking  whether  there  were  additional  aspects  of  the  lETMs  that  participants 
wanted  to  comment  about,  we  received  a  detailed  suggestion  from  an  experienced 
maintainer.  He  believes  that  lETMs  for  novices,  especially  in  a  reduced  workforce 
scenario  should  include  either  video  or  short  animations.  He  noted  that  “either  a  video 
clip  or  some  sort  of  animation  to  show  how  it  slides  in  and  especially  certain  parts  that 
you  don’t  just  -  you  know,  you  look  at  it  and  o.k.  you  put  it  in  there  -  and  you’re  like  how 
do  I  put  it  in  there?  Take  an  EDU  for  example,  drive  wrench,  you  kinda  have  to  tilt  them 
a  little  bit,  pull  and  tilt,  and  stuff  like  that.  But  it  would  definitely  take  interaction  between 
guys  who  have  done  it  before  and  they  know  the  little  tricks  to  getting  parts  in  and  out. 
That  would  be  helpful  to  the  younger  sailors  -  the  more  inexperienced  that  don’t  know 
how  to  do  it.  Cause  if  it’s  just  them  in  the  pit  and  they  say  put  this  in.  Well,  easier  said 
then  done.  That’s  when  you  need  the  other  person  there... .You  have  to  have  someone 
else  there.” 

V.  Conclusions  and  Recommendations 

After  noting  reasons  we  believe  are  associated  with  the  results  of  this  study,  we  make 
four  recommendations  for  the  U.S.  Navy  (and  the  overall  DoD  community)  to  consider 
seriously  if  their  objectives  for  force  reduction,  adequate  operational  readiness,  and 
reduced  training  for  military  maintainers  are  to  be  realized  successfully. 
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Positive  responses  to  mobile  lETMs 


It  appears  that  overall  reaction  was  very  positive  to  use  of  mobile  lETMs  for  aircraft 
maintenance.  Of  course,  maintainers  who  participated  in  this  study  are  on  the  flightline 
daily  trying  to  use  paper  documentation  and  their  current  PED  system  for  F/A-1 8  work 
and  are  finding  this  very  difficult  to  do.  Many  of  them  also  recall  the  difficulties  of  doing 
maintenance  with  these  support  materials  while  on  a  carrier. 


Presented  with  an  alternative  that  gives  each  person  the  support  they  might  need, 
maintainers  found  the  prototype  IETM  system  we  tested  very  attractive.  During  the 
baseline  phase  of  this  study,  we  documented  the  difficulties  of  using  paper 
documentation.  These  difficulties  included  carrying  heavy  books  to  the  aircraft,  having 
no  place  to  set  the  books  while  working,  and  having  pages  flapping  in  the  wind.  This 
latter  problem  tied  up  the  maintainer’s  hands  while  he  kept  his  place  in  the  manual 
instead  of  keeping  his  hands  free  to  do  repairs.  We  also  documented  the  strong  positive 
reactions  that  maintainers  had  to  the  prototype  mobile  IETM  system  used  in  this  study. 
(See  video  clips  showing  the  same  maintainer  doing  task  with  paper,  with  IETM  and 
debriefing  on  the  accompanying  CD  for  an  example).  During  the  debriefs  the 
maintainers  also  commented  about  problems  with  current  lETMs.  For  example,  one 
participant  stated: 

“Myself,  I  prefer  to  use  a  pub.  I  prefer  b  use  paper  pubs  and  when 
they  don’t  have  them  then  I  navigate  the  lETMs.  There’s  a  few  pubs 
out  there.  But  unfortunately  not  enough.  lETMs  are  pretty  hard  to 
navigate  around.  And  the  biggest,  the  biggest  problem  is  that  you  can’t 
get  into  a  specific  area.  You  got  to  be  exact.  Sometimes  that’s  kind  of 
hard  to  do.  You  don’t  know  the  exact  word,  but  you  know  exactly  what 
you’re  thinking  about.  You  know  what  you  want  to  go  look  for  but  you 
get  used  to  calling  it  another  thing  and  then  lETMs  refers  to  as  a 
completely  different  thing.  And  it’s  hard  to  get  to  it.  Eventually,  you  do, 

20  minutes  later.  After  the  missions  already  lost  and  the  pilots  back  in 
the  room.  I’m  just  kidding,  it’s  not  that  bad.  But  it  could  use  some  major 
improvements.  I  don’t  know  too  many  people  that  are  extremely  happy 
the  way  lETMs  navigates.  It’s  a  great  concept  it  just  needs  some 
work.” 
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The  IETM  prototypes  we  tested  helped  to  document  problems  that  maintainers  have  in 
a  real  work  setting.  Some  of  these  problems  are  related  to  differences  in  the 
maintainers’  experience  level.  Maintainers’  performance  and  preferences  for  different 
interfaces  and  for  different  kinds  of  information  were  also  documented.  Given  that  this 
is  a  study  of  seven  people  doing  one  task,  we  are  cautious  about  generalizing.  We  are 
confident,  however,  given  our  results  and  the  issues  found  in  the  literature  that  we  can 
make  recommendations  for  improvements  in  at  least  four  areas.  IETM  interface  and 
lETMs  use  with  mobile  systems  require  further  research  via:  (1 )  prototyping  and  testing 
lETMs  with  adaptive  options  for  users  of  different  experience  levels;  (2)  doing  a  study  of 
multi-user  (help  desk)  prototypes  to  understand  the  interplay  between  the  IETM  and 
remote  expert  help;  (3)  developing  and  testing  of  a  larger  scale  IETM  effort,  including 
the  whole  system,  especially,  the  display  and  I/O  devices  to  really  address  the  whole 
list  of  IETM  insufficiencies  identified  in  this  study;  and  (4)  prototyping  and  testing  of 
lETMs  to  support  magnification  and  tracing  and  orientation  issues  for  large  schematics. 
We  discuss  each  briefly  below. 

Adaptive  options 

Designing  and  conducting  a  comparative  study  of  IETM  interfaces  for  novice  versus 
experienced  users  needs  to  be  done.  A  key  part  of  such  a  study  would  include  addition 
of  multi-media  support  for  novices.  This  seems  important  since  less  experienced 
maintainers  appear  to  need  additional  support  to  conduct  tasks  that  may  render  lETMs 
whatever  the  interface  to  be  insufficient.  In  fact,  one  of  the  most  experienced 
maintainers  at  China  Lake  suggested  during  his  debriefing  session  that  the  Navy  should 
build  and  test  prototypes  that  add  to  the  IETM.  He  commented  that” 

“...either  a  video  clip  or  some  sort  of  animation  to  show  how  it 
slides  in  and  especially  certain  parts  that  you  don’t  just  -  you  know 
you  look  at  it  and  ok  you  put  it  in  there  -  and  you’re  like  how  do  I 
put  it  in  there.  Take  an  EDU  for  example,  you  kinda  have  to  tilt 
them  a  little  bit  and  stuff  like  that.  But  it  would  definitely  take 
interaction  between  guys  who  have  done  it  before  and  they  know 
the  little  tricks  to  getting  parts  in  and  out.  That  would  be  helpful  with 
the  younger  sailors  -  the  more  inexperienced.  Right.  That  don’t 
know  how  to  do  it.  Cause  if  it’s  just  them  in  the  pit  and  they  say  put 
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this  in.  Well,  easier  said  than  done.  That’s  when  you  need  the  other 
person  there.  If  you  have  another  person  there,  it’s  kind  of  like  you 
don’t  really  need  the  book,  the  other  things,  right  in  their  face.  You 
have  someone  else  there.” 

Where  even  the  multi-media/enhanced  lETMs  are  insufficient,  then  there  may  be  a 
need  for  human  assistance.  This  issue  is  addressed  next. 

Multi-user  IETM  support  systems 

To  empirically  determine  the  information  needs  of  personnel  who  have  minimal  training 
and  are  seeking  assistance  from  a  remote,  more  expert  maintainer  would  require  task 
analysis  of  at  least  two  tasks  that  the  Navy  views  very  critical.  The  task  analysis  effort 
could  include  documentation  of  the  nature  and  quality  of  task  dialogue  using  an 
extended  version  of  the  coding  system  CMU  devised  in  previous  research  (supported 
by  DARPA  and  NSF).  Using  this  coding  scheme  we  could  provide  DoD  with  analyses 
that  focus  on  1 )  the  number  and  nature  of  novice  worker’s  questions,  2)  references  to 
objects  in  the  environment,  and  3)  experts’  use  of  uncommon  domain-specific  terms. 
Also,  this  task  analytic  effort  would  provide  an  information  about  the  circumstances  and 
physical  context  associated  with  human  interventions.  This  research  could  help  to 
inform  the  design  of  systems  or  doctrine  to  improve  performance  of  novice  maintainers. 

Interaction  of  platform  and  user  reactions  to  lETMs 

Two  major  factors  in  the  use  of  lETMs  for  maintainers  are  the  viewability  and  the  hands¬ 
free  access  to  information.  A  systematic  heuristic  evaluation  needs  to  be  conducted  in 
variable  physical  settings  to  determine  the  effects  of  using  small  tablet  style  and  head- 
worn  display  technologies  to  view  and  navigate  lETMs.  Also,  multi-modal  interfaces 
and  associated  alternative  I/O  devices  need  to  be  tested.  Expected  outcomes  of  such 
an  evaluation  study  would  be  1)  task  analysis  and  baseline  information  about  use  of 
paper  documentation  for  inspection  and  repair  tasks,  and  2)  comparative  performance 
and  preference  data  for  use  of  different  display  technologies  that  impact  the  amount  of 
screen  real  estate,  the  resolution  of  information  (VGA  versus  SVGA),  the  fonts  and  font 
sizes,  and  other  relevant  factors  that  might  influence  usability  of  lETMs  in  operational 
settings.  The  multi-modal  interface  development  and  testing  with  different  I/O  devices 
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would  provide  some  factual  information  about  what  works  and  what  is  feasible  now  and 
in  the  next  five  years.  The  results  of  this  study  could  be  used  to  inform  major  training 
and  acquisition  decisions. 

Supporting  viewing  of  large  schematic  images 

Being  able  to  keep  track  of  location  and  orientation  of  a  wiring  diagram  or  other  large 
schematic  image  with  any  resource  other  than  paper  is  a  difficult  challenge.  Providing 
viable  performance  support  to  novice  avionics  personnel  or  generalized  maintainers 
who  need  this  kind  of  information  from  online  documents  is  infeasible  today. 

Prototyping  and  field  studies  of  solutions  to  rapidly  find  the  means  to  accomplish  this 
kind  of  support  seems  essential  at  this  time. 

In  sum,  while  the  promise  of  high  level  lETMs  for  Navy  maintenance  is  there,  much 
applied  research  is  needed  to  realize  this  promise. 
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Appendix  I 
ONR  IETM  Study 

Initial  Questionnaire 

August/September,  2000 
Participant!  D: _ 


The  results  from  this  study  will  be  used  to  develop  improved  Interactive  Electronic  Technical 
Manuals  (lETMs)  to  support  vehicular  maintenance  workers  in  the  future. 

To  help  us  better  interpret  your  answers  to  the  questions,  we  need  to  ask  a  few  questions  about 
your  background. 

Please  answer  the  questions  based  on  your  knowledge  and  experience  from  your  current  and 
previous  training  and  experience.  Please  read  and  answer  all  of  the  questions.  If  you  wish  to 
comment  on  any  questions  or  qualify  your  answers,  please  use  the  space  provided. 

Your  answers  will  be  held  in  strict  confidence.  Your  name  will  be  used  for  administrative  purposes 
only. 

Thank  you  for  your  help. 

Dr.  Jane  Siegel 

Human  Computer  Interaction  Institute 
School  of  Computer  Science 
Carnegie  Mellon  University 
Pittsburgh,  Pennsylvania  15213 

SECTION  I. 


1 .  Name: _  2.  Age: _  3.  Rank/position: _ 

4.  When  did  you  become  an  F-18  mechanic?  (Please  specify  year  and  month): _ 

5.  How  long  have  you  been  at  your  current  assignment?  (Please  specify  in  years/months): _ 

6.  How  long  have  you  been  doing  aircraft  repairs?  (Please  specify  in  years/months): _ 

SECTION  II. 

1 .  Did  you  receive  computer  training  before  taking  your  current  job?  (Please  circle):  Yes  No 

If  yes,  please  circle  all  the  appropriate  categories: 

Operations  Programming  Word  Processing  Database  Other 

If  other,  please  describe  briefly  below: 

2.  If  you  have  prior  training  on  computers,  are  you  familiar  with  (Please  circle):  Mac  PC  Other 

If  other,  please  describe  briefly  below: 
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3.  On  a  scale  of  1  to  7,  rate  your  level  of  experience  using  computers  (Please  circle): 

1  2  3  4  5  6  7 

None  Moderate  Extensive 

4.  Did  you  receive  prior,  formal  aircraft  maintenance  training  in  the  military?  (Please  circle): 

Yes  No 

If  yes,  please  describe  briefly  below: 

5.  Did  you  receive  other  formal  vehicle  maintenance  training  before  military  service?  (Please  circle): 

Yes  No 

If  yes,  please  describe  briefly  below: 

6.  On  a  scale  of  1  to  7,  please  rate  your  skill  level  for  inspecting  or  repairing  aircraft.  (Please  circle): 

1  2  3  4  5  6  7 

None  Moderate  Extensive 

7.  Have  you  used  any  military  diagnostic  tools  or  online  (computerized)  lETMs  previously?  (Please  circle) 

Yes  No 

If  yes,  please  specify  which  tool(s)  below: 

8.  Do  you  have  20/20  vision  in  both  eyes?  (Please  circle):  Yes  No 

If  no,  please  answer  the  following: 

8a.  Do  you  wear  corrective  lenses  (eyeglasses  or  contact  lenses)  routinely  at  work? 

(Please  circle):  Yes  No 

If  "yes",  please  indicate  nature  of  correction,  e.g.,  near-sighted,  far-sighted,  bifocals,  etc. 
below: 

8b.  Have  you  ever  had  any  eye  problems,  e.g.,  strain,  sensitivity  to  light,  etc. 

(Please  circle):  Yes  No 

If  yes,  please  explain  briefly  below: 


Thank  you  for  your  help. 


Appendix  2 


ONR  IETM  Study 
Post  Questionnaire 
10/2000 

Participant  ID: _ 

Date: _ 

1.  Please  choose  the  best  answer  for  each  of  the  following: 

A.  How  sure  are  you  that  you  completed  the  task  correctly?  (1  =  Not  at  all  sure  and  5  =  Very  Sure) 

B.  How  easy  was  it  to  complete  the  task?  (1  =  Very  difficult;  5  =  Very  easy) 

C.  How  useful  did  you  find  the  Technical  Manual(s)  in  completing  the  task?  (1  =  Not  at  all  useful  and 

5  =  Very  Useful) 

For  each  item  in  question  5,  please  answer  the  "yes”  “no”  choice  and  give  a  brief  explanation  for  any 
items  when  you  checked  “yes ’’(Please  use  the  back  side  of  this  sheet  if  you  need  more  space) 

1.  Did  you  experience  any  problems  with  using  the  Technical  Manuals  when: 

5.1:  indoors  yes  no 

5.2:  outside  yes  no 

5.3:  wearing  safety  glasses  _ yes  _ no 

5.4:  talking  with  others  _ yes  _ no 

5.5:  other  yes  no 

•  Did  you  have  any  difficulties  reading  the  text  in  the  Technical  Manuals?  _ yes _ no 

If  yes,  please  indicate  which  TO,  topic  and  explain  briefly 

•  Did  you  get  confused  about  where  you  were  in  the  Technical  Manuals?  _ yes _ no 

If  yes,  please  explain  briefly 

•  Did  you  have  any  difficulties  finding  the  information  you  need?  _ yes  _ no 

If  yes,  please  explain  briefly 


For  the  Baseline  study,  please  stop  here 
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For  the  online  study,  please  complete: 


8.  a  Please  circle  your  answer: 


Using  web  manual 

Very  Easy 

Very  Difficult 

1 .  Seeing  the  text  in  the  online  application  was 

1 

2 

3 

4 

5 

2.  Seeing  the  graphic  representations  was 

1 

2 

3 

4 

5 

3  Using  the  index  to  find  information  was 

1 

2 

3 

4 

5 

4.  Finding  the  information  1  needed  to  link  to  was 

2 

3 

4 

5 

8.  b  Please  circle  your  answer: 


Using  experimental  IETM 

Very  Easy 

Very  Difficult 

1.  Seeing  the  text  in  the  online  application  was 

1 

2 

3 

4 

5 

2.  Seeing  the  graphic  representations  was 

1 

2 

3 

4 

5 

3  Using  the  index  to  find  information  was 

1 

2 

3 

4 

5 

4.  Finding  the  information  1  needed  to  link  to  was 

2 

3 

4 

5 

9.  Did  you  have  any  difficulties  using  the  buttons?  _ yes  _ no 

If  yes,  please  explain  which  button(s)  and  the  problems  you  found: 


10.  What  do  you  like  best  about  the  IETM  prototypes? 


1 1 .  What  do  you  like  least  about  the  IETM  prototypes? 


12.  What  one  thing  would  you  change  about  the  IETM  prototypes  if  you  could? 


13.  Is  there  anything  else  you  would  like  us  to  know? 


Thank  you  for  your  participation  in  this  study. 
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